


Practical Neural 
Networks (1) 


Part |: 
an introduction to Neural Nets 





compint/page.cfm?pge=6732 


By Chris McLeod and Grant Maxwell 


Artificial Neural Networks (Neural Nets or just ANNs for short) are a 
popular form of Artificial Intelligence (Al). They are based on the 
operation of Brain Cells and many researchers think that they are our 
best hope for achieving true intelligence in a machine. If you’re a fan of 
the TV Series ‘Star Trek’, you’ll know that Mr Data is supposed to have a 


Neural Net brain, as is the robot in the ‘Terminator’ films. 


Although these technological wonders are on 
the cutting edge of research in Computer Sci- 
ence, they are also well within the grasp of 
the enthusiastic amateur. The aim of these 
articles is to introduce this fascinating topic 
in a practical way, which will allow you to 
experiment with your own Neural Nets. 


So, what exactly is 


a Neural Net? 
A Neural Net is a network of connected 
processors. Each individual processor is only 
capable of a very simple mathematical task, 
but when many are connected together com- 
plex behaviour results. The drawing in Fig- 
ure 1 shows a typical neural net. 

To understand the Neural Network let’s 
look at its inspiration - the brain. 


It’s all in your head 


What a layman calls ‘Brain Cells’ are what a 
biologist would refer to as Neurons. These are 
the tiny living cells, which make up our brain 
and nervous system. There are about 100 bil- 
lion of them in a typical human. Each one has 
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Figure |. A typical Neural Net. 


Dendrites (receive signals 
from the outside world or 


4— from other neurons). 


Cell Body or Soma 
(produces a pulse when 


the dendrites have 
received enough input). 
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that the pulses 
travel along). 
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muscles, glands or other 
neurons). This is the 
Output end of the neuron. 
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Figure 2. A Biological Neuron. 
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Figure 3. How a neuron processes information. 


evolved to be an electronic proces- 
sor. If you looked at the Neuron 
under the microscope, it would 
appear as shown in Figure 2. 

The Dendrites receive signals from 
other neurons or the outside world; 
they are the “receiving end” of the 
Neuron. When enough stimulus is 
received by the Dendrites, the Cell 
Body generates a pulse of electricity 
which travels down the Axon to the 
Dendrites of other neurons or to mus- 
cles, stimulating them. 

As a neuron gets more stimula- 
tion through its dendrites, it pro- 
duces more pulses per second as 
shown in Figure 3. Therefore, the 
information in the nervous system is 
frequency modulated. 

Some neurons, called Sensory 
Neurons, collect information from the 
outside world. The Rods and Cones 
in your eyes are Sensory Neurons. 
This stimulus is passed up to the 
brain along the Axons of the Sensory 
Neurons. There it is processed by 


Figure 4. The basic Artificial Neuron. 
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others, called Interneurons, which 
are connected only to other neurons. 
The result of all this is passed to a 
final set, called Motor Neurons, the 
axons of which are connected to 
muscles and glands. 

In this way we can read informa- 
tion in from the world, process it and 
act upon it. These processes are 
responsible for everything, from sim- 
ple reflexes to consciousness itself. 


The Birth of the ANN 


In 1943 two researchers named War- 
ren McCulloch and Walter Pitts pub- 
lished a paper which started the sci- 
ence of Artificial Neural Nets. It pro- 
posed a simple model of the neuron, 
and Artificial Networks are based on 
it (in improved form), to this day. The 
basic idea is shown in Figure 4. 
The diagram shows four inputs 
labelled i, to i4. These are like the 
dendrites of the biological version 
and connect to the outside world or 






Threshold 


020324 - 1 - 14 





to other neurons. The inputs are “weighted” 
by four weights w, to w4. These weights rep- 
resent the strength of the connections to the 
cell body in the real neuron. There may, of 
course, be any number of inputs in reality. 

The system works like this. First, each 
input is weighted by multiplying it by its 
associated weight and then the result of all 
the weighted inputs are added together: 


Sum = (i X W4) + (i2 X w2 ) + (i3 x w3) + (ig 
X Wa) 


Then, if this sum (which in the biological 
sense is the total stimulus going into the neu- 
ron) is above a pre-set threshold, the neuron 
produces a “1” output; if not it produces a 
“0”. So, let's say the threshold is 0.5: 


If Sum > 0.5 then Output = 1 
If Sum < 0.5 then Output = 0 


This is a bit like the real neuron which sends 
out a pulse when its stimulus is big enough. 
Let’s try a little example to make sure 
everything’s clear so far. Suppose we have a 
neuron with inputs and weights as shown in 
Figure 5, what would its outputs be? Calcu- 
late the output from the neuron in the draw- 
ing, assuming a threshold of 0.5: 
Sum = (0.1 x 0.5) + (0.5 x 0.2) +(0.3 x 0.1) = 0.18 
Since 0.18 is less than the threshold, Output = 0 


So what’s this good for? 


OK, so we have a model of a neuron, but what 
is it good for? Well, let's take a visual pattern 
like that shown in Figure 6a. And let’s say 
we want a device which can recognise it. We 
can code the dark pixels as ones and the light 
pixels as zeros. If we connect up our neuron 
as shown in Figure 6b, it will produce a “1” 
output whenever it “sees” this pattern. You 
can probably see that even if the pattern isn’t 
perfect — if the dark pixels aren't quite ones 
and the light ones aren't quite zeros, the neu- 
ron will still recognise the pattern (we say 
that the neuron is noise tolerant). 


0.1 0.5 
Output 

0.5 o 

0.1 

0.3 
Inputs Weights 020324-1-15 
Figure 5. Aworked example. 
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But if we put in a completely different pat- 
tern, like that shown in Figure 6c, the neu- 
ron won't recognise it. 

Recognising patterns, even when they’re 
not perfect, is one of the most basic attributes 
of intelligence. You have to be able to recog- 
nise a mate or food or a predator to survive, if 
you're an animal. 

Hopefully, after some thought, you'll be 
able to see the importance of pattern recog- 
nition. Neural nets are arguably the best tech- 
nique for doing this sort of task. They can also 
recognise patterns in time, if they are sam- 
pled and fed into the network. 


Now Learn! 


You'll also see from the example above, that 
the set up of the weights is critical to the 
functioning of the network. It’s the weights 
that determine what pattern the network 
recognises, and setting the weights is the 
main task of the neural net program. 

In our example above, these weights have 
been set by common sense and for a very 
simple task like this, that is possible. But for 
complex tasks like recognising hundreds of 
patterns with a network of neurons, it’s 
impossible. We also want the weights to be 
set automatically, without a human operator 
having to interfere; after all, we all learn by 
ourselves without any external help. 

Enter the “Learning” or “Training” Algo- 
rithm. This is the part of the neural net which 
sets up the weights to automatically recog- 
nise a pattern. To give a very simple example 
of how this might work, consider an algo- 
rithm for teaching the neuron above to recog- 
nise its pattern: 


m 


. If Output is correct then do nothing. 

2. If Output is high but should be low, then 
decrease the weights attached to high 
inputs. 

3. If Output is low but should be high, then 

increase the weights attached to big 

inputs. 


The weights usually start off as random num- 
bers. The other articles in this series are about 
the various methods of training the network. 

Two other points are worth mentioning 
here. Firstly, by setting up the weights the 
way you want, you can make the network 
behave as (almost) any type of logic gate. So 
don’t think of the neural net just as a recog- 
niser — it’s more flexible than that. Actually 
it’s more like a trainable combinational logic 
circuit. Secondly, rather than using a pure 
binary output as shown in the examples 
above, we now generally use a mathematical 
function known as the “Sigmoid” function in 
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Output = 1 
(Pattern recognised) 











Sum = (1x1)+(0x0)+(1x1)+(0x0) = 2 
Sum is > threshold ~. Output = 1 











Output = 0 
(Pattern not recognised) 





Sum = (0x1)+(1x0)+(0x1)+(1x0) = 0 
Sum < threshold ~. Output = 0 











c. The network won't recognise a different pattern. 
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Figure 6. Using the neuron for pattern recognition. 





In the threshold case, the output 
changes suddenly from a “zero” to a 
“one”, however in the case of the 
sigmoid this happens gently. This 
helps the neuron to express 
uncertainty 
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Figure 7. The Binary and the Sigmoid neuron. 
the output of our neurons. This is: 
Output = 
1+ e Sum 
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No matter how you 
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truth table shown 
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Fig.8. A single neuron cannot simulate an EXOR gate. 
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Inputs First layer of neurons 
(internal layers are called 
“hidden” layers). 


Figure 9. A multilayer neural network. 
So, in the example shown in Figure 


5 the output of the neuron, using the 
sigmoid function would be: 


zal = 9 5 
142°" 


Output = 


Actually, this function always pro- 
duces an output between zero and 
one; therefore the output is like the 
binary threshold. So why use it? Well, 
its advantage comes in the circum- 
stance of two patterns being very 
similar. It allows the neuron to 
express uncertainty by having an out- 
put which is “in-between” zero and 
one. The network is saying “I’m not 
quite sure”. Figure 7 shows the idea. 


Networks of neurons 


Neural Nets had gone from strength 
to strength in the years since McCul- 
loch and Pitts first proposed their 
model. But in 1969 a book was pub- 
lished which attacked many of the 
ideas behind them. 

The book was “Perceptrons” by 
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Marvin Minsky and Seymour Papert. 
It pointed out a flaw in the basic neu- 
ron. The problem was this: No matter 
what values you set the weights of a 
two input neuron, it will never be 
able to simulate an Exclusive OR 
(XOR) gate as shown in Figure 8. 

This, along with other arguments 
in the book, held up progress in 
neural nets for many years. Most 
researchers left the field, feeling that 
if a neuron couldn't even fulfil this 
simple task, then it wasn’t worth 
researching. 

Actually, what was needed to 
overcome this problem and others 
like it, was not a single neuron, but 
a network of them. In fact, a Russian 
mathematician called Kolmogorov 
has already shown that a neural net- 
work with three layers as shown in 
Figure 9 could solve any problem of 
this kind. 

However, although this solution 
became known, it wasn’t until 
1982/83 that neural nets really took 
off as a research area again. At that 
time, the Back Propagation Algo- 
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Figure ||. A hardware implementation of a neuron. 
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Figure 10. An example of a neural network. 


rithm (to be described in part 2) became well 
publicised, cheap computing became avail- 
able to the masses and John Hopfield pub- 
lished his famous paper (to be covered in part 
3 of this series). 

The neurons in the network work exactly 
the same way as the ones already covered. 
Let’s do an example to demonstrate with sig- 
moid neurons, Figure 10. 

Calculate the output from this network 
assuming a Sigmoid Squashing Function. 


Input to neuron 1 = (0.2 x 0.5) + (0.4 x 0.8) = 
0.42. 


Output = -e 0.603 
0.42 


1+e 


Input to neuron 2 = (0.2 x 0.1) + (0.4 x 0.9) = 
0.38. 


Output = enan = 0.594 
0.38 


1l+e 


Input to final neuron = (0.594 x 0.7) + (0.603 x 
0.5) = 0.717. 


= 0.672 


Final output = 
-0.717 


1+e 


Make your own! 


You can implement neural nets in hardware, 
either analogue or digital. A block diagram 
of an analogue implementation is shown in 
Figure 11. 

Three simple amplifiers weight the inputs; 
these have to have electronically adjustable 
gains if the network is to learn in real time. 
The output is then summed and the thresh- 
old is applied by a comparator. 

This type of circuit has been built using 
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discrete components like Op Amps and as 
part of an Integrated Circuit. The disadvan- 
tage of this approach is that it requires a lot 
of components, should there be many neu- 
rons. 

One can fabricate digital circuits in exactly 
the same way. The weight amplifiers would 
be replaced by digital multipliers, the sum- 
ming amplifier by an adder and the compara- 
tor by a decoder which would fire if the 
binary output of the adder was bigger than a 
threshold value. Digital neural nets are 
wasteful of resource, however, since there 
require binary multipliers which are usually 
large and complex circuits. 

By far the most common way to implement 
a neural net is to simulate it in software. This 
allows the programmer to quickly change 
parameters like the number of neurons and 
how they are connected without costly 
rewiring and means that large numbers of 
neurons can be connected easily. 

The software approach has many advan- 
tages for the amateur experimenter and it is 
relatively easy to program a neural net in any 
high level language and many assembly lan- 
guages (especially those with multiply com- 
mands). Given these advantages, it is this 
approach that we will pursue in the rest of 
this series. 


Software implementation 


So how might we code the neural net in soft- 
ware? The easiest way is to store all the nec- 
essary parameters in arrays. The inputs to, 
and outputs from, the network can be stored 


o(1,1) 


O(1,2 
ve W(1,2,2) 






W(2,1,D 


W(2,1,2) 
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Figure | 2. Storing the weights in a typical network. 
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Figure 13. A more flexible way of coding the network. 


in one array and the weights in 
another. 

For example, we could store all 
the weights in a three dimensional 
array: 


Weights [layer number, neu- 
ron_number, connection_num- 
ber ] 


Listing 1. Network output calculation 


FOR L = 1 TO number of layers 


FOR n = 1 TO number of neurons _in biggest_layer 
FOR c = 1 TO number weights in biggest_layer 


O(L+1, n) = 
NEXT c 


O(L+1, n) + O(L, c) * W(L, n, C) 


O(L+1, n) = 1/(1+exp(-1*0O(L+1,n) )) 


NEXT n 
NEXT L 


And the outputs (and the inputs) 
from each neuron in a two dimen- 
sional array: 


Output[ layer number, 
ron_number]. 


neu- 


Then the network generated would 
have the form shown in Figure 12. 


Listing 2. Alternative output programming 


FOR t = 1 + number of inputs TO number of neurons 


FOR f = 1 TO number of _ neurons 
O(t) = O(t) + O(f) * W(f,t) 


NEXT f 


O(t) = 1/(1 + exp(-1 * O(t))) 


NEXT t 
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COMPUTING 


Notice that O(1,1) and O(1,2) are 
the inputs to the network and O(3,1) 
is the output. We can make the com- 
puter calculate the output from such 
a network, with the simple algo- 
rithm: 

1. Initialise all weights and outputs 
to zero. 

2. Set up the inputs, in this case 
O(1,1) and O(1,2), to the values 
you want. Layer one is the input 
to the net. 

3. Set up the weights used in the 
network to the values you want. 

4. Calculate the output from the net- 
work: see Listing 1. 


There are actually many different 
ways of coding the network like this. 
Another, perhaps more flexible way 
is to code the weights like this: 


weight 
[neuron _weight_is coming fr 
om, 
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neuron weight _is_ going to] 


and just number each neuron’s out- 
put (and input) sequentially: 


output [neuron_number] 


This would give a network of the 
type show in Figure 13. 

In this case O(1) and O(2) are the 
inputs and O(5) is the output. This 
time the algorithm to calculate the 
output is: 


1. Initialise all weights and outputs 
to zero. 

2. Set up the inputs, in this case O(1) 
and O(2), to the values you want. 

3. Set up the weights used in the 
network to the values you want. 

4. Calculate the output from the net- 
work, see Listing 2. 


Some languages (like C and C++) 
allow you to dynamically allocate the 


array sizes you want, this can be useful in 
complex networks which change their size. 
Other languages like FORTRAN and MATLAB 
allow matrix multiplication which make life 
easier since all the network equations can be 
written as matrix operations. 

Object Orientated languages allow you to 
define Neurons and Networks as objects and 
inherit the parameters of your neurons into 
your networks. 

In low level languages, the weights and 
outputs may be stored directly in memory 
and routines written to access and manipu- 
late them. 

Calculating the output of the network is 
known as the Forward Pass. The next stage 
is to change the weights so that the network 
learns to do better on the next forward pass; 
this is how the network is trained and is 
called the Reverse Pass. 

In the second article we'll have a look at 
training the network, starting with the most 
popular learning algorithm — the Back Prop- 
agation Algorithm. 
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